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ABSTRACT 

The  number  and  type  of  unmanned  vehicles  sought  in  military 
operations  continues  to  grow.  A  critical  consideration  in 
designing  these  systems  is  identifying  interface  types  or 
interaction  schemes  that  enhance  an  operator's  ability  to  supervise 
multiple  unmanned  vehicles.  Past  research  has  explored  how 
interface  types  impact  overall  performance  measures  (e.g.  mission 
execution  time),  but  has  not  extensively  examined  other  human 
performance  factors  that  might  influence  human-robot  interaction. 
Within  a  dynamic  military  environment,  it  is  particularly 
important  to  assess  how  interfaces  impact  an  operator’s  ability  to 
quickly  adapt  and  alter  the  unmanned  vehicle’s  tasking.  To  assess 
an  operator’s  ability  to  confront  this  changing  environment,  we 
explored  the  impact  of  interface  type  on  task  switching.  Research 
has  shown  performance  costs  (i.e.  increased  time  response)  when 
individuals  switch  between  different  tasks.  Results  from  this  study 
suggest  that  this  task  switching  effect  is  also  seen  when 
participants  controlling  multiple  unmanned  vehicles  switch 
between  different  strategies.  Results  also  indicate  that  when 
utilizing  a  flexible  delegation  interface,  participants  did  not  incur 
as  large  a  switch  cost  effect  as  they  did  when  using  an  interface 
that  allowed  only  the  use  of  fixed  automated  control  of  the 
unmanned  vehicles. 

Categories  and  Subject  Descriptors 

H. 1.2  [Models  and  Principles]:  User/Machine  Systems  -  Human 
factors. 

General  Terms 

Measurement,  Performance,  Design,  Experimentation,  Human 
Factors. 

Keywords 

Automation,  Delegation,  Human-Robot  Interaction,  Playbook, 
Unmanned  Vehicles,  Interruption,  Task  Switching 

I.  INTRODUCTION 

Many  different  types  of  unmanned  vehicles  (UVs)  are  being 
developed  for  use  in  aerial,  ground,  and  underwater  environments. 
Next  generation  UV  systems  such  as  the  United  States  Army’s 
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Future  Combat  Systems  (FCS)  will  incorporate  numerous  ground 
and  air  UVs,  with  the  type  of  UV  and  team  size  being 
reconfigurable  components  tailored  to  specific  combat  missions 
[3],  In  addition,  military  objectives  are  focused  on  allowing  a 
small  number  of  personnel  to  supervise  a  large  number  of  UVs. 
These  trends  in  UV  development  have  created  a  need  for 
understanding  how  operator(s)  can  effectively  control  a  large 
number  of  UVs  of  varying  types  and  capabilities. 

If  supervisors  are  going  to  be  responsible  for  overseeing  multiple 
UVs,  it  is  plausible  that  they  would  employ  sub-sets  of  UVs  to 
accomplish  different  objectives.  If  a  supervisor  has  to  switch 
between  UV(s)  performing  the  same  or  different  objectives,  what 
impact  will  that  have  on  their  performance?  Previous  research 
indicates  that  such  task  switching,  or  interruptions,  can  be 
disruptive,  particularly  if  there  are  insufficient  environmental  cues 
to  allow  timely  resumption  of  the  interrupted  task  [2,  15].  In 
addition  to  switching  between  tasks,  research  has  shown  that 
switching  between  vaiying  levels  of  automation  can  result  in  both 
positive  and  negative  performance  [14].  Therefore,  switching 
between  different  objectives  or  between  different  levels  of 
automation  may  positively  or  negatively  impact  a  supervisor’s 
performance. 

When  managing  UV  systems  a  supervisor’s  ability  to  effectively 
react  to  a  changing  environment  therefore,  may  be  contingent 
upon  the  operator  having  to  (1)  switch  levels  of  automation,  or  (2) 
switch  task  actions.  Level  of  automation  refers  to  the  full  or 
partial  replacement  of  a  task  (i.e.  function)  previously  carried  out 
by  the  human  supervisor.  For  example,  UV(s)  supervised  by 
waypoint-to-waypoint  action  would  represent  a  lower  automation 
level  than  a  UV  supervised  by  pre-programmed  behaviors  such  as 
“patrol  border’’.  A  task  action  is  defined  by  the  strategies 
necessaiy  to  achieve  a  higher  level  objective.  For  example,  in  the 
game  of  capture-the-flag,  the  highest  level  objective  is  to  win  the 
game  -  by  capturing  the  opponent’s  flag  and  returning  back  to 
own  side.  To  fulfill  this  objective  a  player  must  have  both  a 
defensive  (protect  own  flag)  and  offensive  strategy  (capture 
opponent  flag). 

From  these  definitions  of  levels  of  automation  and  task  action 
(from  here  on,  referred  to  as  strategy)  it  is  possible  to  describe 
situations  in  which  a  switch  would  occur.  For  example  a  switch 
between  levels  of  automation  would  occur  if  a  supervisor  was 
operating  a  UV  in  a  waypoint-to-waypoint  mode  and  switched  to 
more  automated  condition  such  as  “patrol  border’’.  A  strategy 
switch  would  occur  if  the  supervisor  went  from  an  offensive 
strategy  to  a  defensive  strategy  -  or  vice-versa.  A  no-switch 


26 


Report  Documentation  Page 


Form  Approved 
OMB  No.  0704-0188 


Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 


1.  REPORT  DATE 

2006 


2.  REPORT  TYPE 


4.  TITLE  AND  SUBTITLE 

Human  Control  of  Multiple  Unmanned  Vehicles:  Effects  of  Interface 
Type  on  Execution  and  Task  Switching  Times 

6.  AUTHOR(S) 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Naval  Research  Laboratory, Navy  Center  for  Applied  Research  in 
Artificial  Intelligence  (NCARAI),4555  Overlook  Avenue 
SW, Washington, DC, 20375 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 


3.  DATES  COVERED 

00-00-2006  to  00-00-2006 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

8.  PERFORMING  ORGANIZATION 
REPORT  NUMBER 


10.  SPONSOR/MONITOR'S  ACRONYM(S) 

11.  SPONSOR/MONITOR'S  REPORT 
NUMBER(S) 


12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

HRI’06,  March  2P4,  200a,  Salt  Lake  City,  Utah 

14.  ABSTRACT 

The  number  and  type  of  unmanned  vehicles  sought  in  military  operations  continues  to  grow.  A  critical 
consideration  in  designing  these  systems  is  identifying  interface  types  or  interaction  schemes  that  enhance 
an  operator:  s  ability  to  supervise  multiple  unmanned  vehicles.  ;ast  research  has  explored  how  interface 
types  impact  overall  performance  measures  (e.g.  mission  execution  time),  but  has  not  extensively  examined 
other  human  performance  factors  that  might  influence  human-robot  interaction.  Within  a  dynamic 
military  environment,  it  is  particularly  important  to  assess  how  interfaces  impact  an  operators  ability  to 
Buickly  adapt  and  alter  the  unmanned  vehicle  :s  tasking.  To  assess  an  operator  :s  ability  to  confront  this 
changing  environment,  we  explored  the  impact  of  interface  type  on  task  switching.  Research  has  shown 
performance  costs  (i.e.  increased  time  response)  when  individuals  switch  between  different  tasks.  Results 
from  this  study  suggest  that  this  task  switching  effect  is  also  seen  when  participants  controlling  multiple 
unmanned  vehicles  switch  between  different  strategies.  Results  also  indicate  that  when  utilizing  a  flexible 
delegation  interface,  participants  did  not  incur  as  large  a  switch  cost  effect  as  they  did  when  using  an 
interface  that  allowed  only  the  use  of  fixed  automated  control  of  the  unmanned  vehicles. 

15.  SUBJECT  TERMS 


16.  SECURITY  CLASSIFICATION  OF: 

17.  LIMITATION  OF 

18.  NUMBER 

19a.  NAME  OF 

ABSTRACT 

OF  PAGES 

RESPONSIBLE  PERSON 

a.  REPORT 

unclassified 

b.  ABSTRACT 

unclassified 

c.  THIS  PAGE 

unclassified 

Same  as 
Report  (SAR) 

7 

condition  would  occur  if  a  supervisor  contentiously  used  the  same 
level  of  automation  throughout  -  e.g.  always  operated  the  UV  in 
waypoint-to-waypoint  mode.  Likewise,  if  the  supervisor 
continuously  used  and  re-engaged  a  defensive  strategy,  then  there 
would  be  no  strategy  switch. 

The  impact  of  switching  between  different  strategies  when 
utilizing  varying  levels  of  automation  has  not  been  extensively 
examined.  In  particular  research  has  not  examined  this  impact  as  a 
function  of  interface  types.  Interface  type  can  be  described  by  the 
interaction  possibilities  afforded  to  a  supervisor.  For  example,  an 
interface  could  constrain  an  operator’s  ability  to  control  a  UV,  by 
only  providing  a  waypoint-to-waypoint  control.  Alternatively  a 
more  flexible  interaction  scheme  would  allow  the  supervisor  the 
ability  to  choose  between  waypoint  control  and  automated 
control. 

The  impact  of  interface  type  on  overall  operator  performance 
measures  (e.g.  mission  execution  time)  has  yielded  important 
information  about  interface  design  and  the  potential  effect  on  task 
switching.  Specifically,  interfaces  that  allow  operators  to  task 
robots  flexibly  at  different  levels  of  automation,  or  delegation 
interfaces  have  been  found  to  be  especially  useful.  Delegation 
interfaces  enable  the  human  to  set  an  objective  and  decide 
whether  to  automate  (or  not)  tasks,  dynamically  during  system 
operations  [9],  The  Playbook™  is  a  specific  form  of  a  delegation 
interface  that  is  similar  to  the  sports  playbook  concept  -  where 
there  is  an  approved  book  of  plays,  and  selections  of  those  plays 
are  performed  by  the  team  leader  and  executed  by  the  team 
members.  To  implement  Playbook™  the  simulation  environment 
RoboFlag  (Figure  1)  provides  the  capability  to  command 
simulated  robots  [the  rest  of  the  paper  will  use  the  term  robots 
rather  than  simulated  robots],  individualy  or  in  groups,  using 
either  manual  control  (providing  designated  endpoints  for  robot 
travel)  or  automated  control  (higher  level  behaviors, 
preprogrammed)  such  as  “patrol  border”. 


Figure  1.  RoboFlag  “play”  condition  interface. 

In  a  series  of  experiments  using  Roboflag,  Parasuraman,  Galster, 
Squire,  Furukawa,  and  Miller  [13]  showed  that  the  type  of 
interface  available  to  operators  had  significant  effects  on  overall 
human-robot  performance.  They  found  that,  compared  to  a 
restricted  interface  in  which  only  automated  control  was  available, 
mission  execution  times  were  shorter  when  the  operator  was  able 
to  flexibly  use,  at  times  of  their  own  choosing,  either  manual  (i.e. 
waypoint)  control  of  robots  or  automated  control  (e.g.,  such  as 
"patrol  border").  This  benefit  was  attributed  to  the  operator 


recognizing  conditions  where  the  automation  was  “brittle”  and 
needed  to  be  over-ridden  by  tasking  the  robot(s)  in  a  different 
way.  Benefits  of  the  delegation  interface  for  mission  accuracy 
were  also  found,  and  while  subjective  workload  did  increase  with 
the  use  of  delegation,  the  increase  was  not  substantial. 

Parasuraman  et  al.  [13]  used  overall  mission  execution  time  as  a 
performance  metric  and  did  not  explore  the  microstructure  of 
human-robot  interaction  times  as  a  function  of  interface  type.  One 
possibility  is  that  different  interfaces  have  different  effects  on  the 
costs  of  switching  between  strategies.  Many  basic  cognitive 
psychology  studies  indicate  that  when  participants  have  to  switch 
from  performing  one  cognitive  operation  on  a  set  of  stimuli  (e.g., 
addition)  to  another  operation  (e.g.,  addition),  that  there  is  an 
increase  in  reaction  time  (RT)  following  the  switch,  compared  to 
no-switch  conditions  [1,8,10,11,12  ].  The  task-switching 
paradigm  has  also  been  used  with  more  complex  tasks  in  which 
operators  have  to  return  to  a  primary  task  following  interruption 
by  a  secondary  task  [2],  It  is  reasonable  to  suppose,  therefore,  that 
there  will  be  a  switch  cost  in  a  multi-robot  setting  when 
participants  switch  between  different  strategies  that  they  use  in 
supervising  the  robots.  Moreover,  if  these  switch  costs  are 
present,  how  does  the  type  of  interface  supporting  human  tasking 
of  the  robots  affect  those  switch  costs? 

One  possibility  is  that  switch  costs  increase  more  in  highly 
automated  interaction  schemes  between  the  human  operator  and 
the  robot,  than  with  lower  levels  of  automation  [4],  For  example, 
Crandall,  Goodrich,  Olsen,  and  Nielsen  [4]  estimated  higher 
switch  costs  for  point-to-point  (P2P)  and  multiple-waypoint 
tasking  of  robots  in  a  simulated  navigation  task,  compared  to  less 
automated  interaction  involving  teleoperation.  The  basis  for  this 
expected  performance  was  derived  from  the  interface  efficiency 
(the  increase  in  performance  of  a  neglected  robot  from  human 
interaction)  of  an  interaction  scheme.  Crandall  et  al.  [4]  indicated 
that  these  estimates  may  not  always  be  accurate,  and  that  further 
work  should  be  conducted  to  explore  such  switch  costs.  The 
theoretical  importance  of  these  interface  efficiency  predictions 
concerns  (1)  the  time  necessary  to  regain  awareness  of  the  robot’s 
state,  and  (2)  how  different  interaction  schemes  may  influence 
that  interaction  time. 

Interfaces  that  allow  operators  to  regain  awareness  of  the  robot’s 
state  more  quickly  should  reduce  switch  costs,  while  interfaces 
that  hinder  an  operator’s  ability  to  regain  awareness  of  the  robot’s 
state  should  increase  switch  costs.  Research  with  delegation 
interfaces  suggests  that  providing  an  operator  flexibility  in  making 
decisions  on  how  to  task  or  maintain  awareness  of  a  robot’s  state 
should  reduce  switch  costs  as  well  as  reducing  overall  mission 
execution  time. 

These  proposals  suggest  that  it  would  be  worthwhile  to  re¬ 
examine  the  effects  of  delegation  interfaces  on  human-robot 
teaming  performance  taking  switching  costs  into  account. 
Accordingly,  we  conducted  a  study  on  the  effects  of  delegation 
interfaces  on  human-robot  interaction  using  the  RoboFlag 
simulation.  We  hypothesized  that  different  interface  types  would 
result  in  different  switch  costs.  Second,  following  Crandall  et  al. 
[4],  we  also  reasoned  that  lower  levels  of  automated  control  of 
robots,  such  as  waypoint-to-waypoint  control,  would  incur  lower 
switch  costs  than  when  participants  used  automated  plays.  In 
addition,  based  on  our  previous  results  [13],  we  hypothesized  that 
a  flexible  delegation  interface  would  also  reduce  switch  costs 
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compared  to  restricted  interfaces  in  which  the  operator  only  had 
the  option  of  using  automated  control. 

2.  METHODS 

2.1  Participants 

Twelve  adults,  7  males  and  5  females  aged  18-22  years  (M  = 

20.1  yrs.,  SE  =  0.4  years.)  served  as  paid  participants.  All 
participants  reported  normal  or  corrected  to  normal  visual  acuity. 

2.2  Experimental  Design 

A  within-subjects  design  was  employed  using  interface  conditions 
selected  from  the  different  combinations  of  automation  level  (or 
abstraction)  and  robot  selection  (or  aggregation)  -  see  Figure  2. 
Automation  level  was  defined  as  a  continuum  of  tasking  action  as 
applied  to  individual  or  groups  of  robots.  At  a  low  level,  the 
supervisor  could  use  waypoint-to-waypoint  movement  control  of 
the  robot  (an  even  lower  level  would  involve  telerobotic  control, 
but  in  this  study  only  robots  capable  of  autonomous  movement 
were  considered).  Waypoint  movements  (i.e.  manual  control) 
were  defined  as  the  selection  of  some  number  of  robots  followed 
by  pointing  and  clicking  to  a  desired  location  in  the  field  to  which 
the  robots  would  move.  At  a  higher  task  level,  the  operator  tasked 
single  or  multiple  robots  with  “plays”.  Plays  consisted  of  pre¬ 
planned  continuous  movement  actions  and  reactions  to  events 
(e.g.  a  participant’s  robot  takes  action  to  not  be  tagged  by  an 
opponent  robot).  At  the  highest  task  level,  the  operator  had  the 
option  to  use  “superplays,”  which  consisted  of  a  mix  of  different 
plays  using  more  than  one  robot.  The  second  dimension,  robot 
selection,  was  defined  as  the  number  of  robots  to  which  particular 
tasks  were  assigned.  “Individual”  robot  selection  referred  to 
commands  given  to  an  individual  robot.  “All”  robot  selection 
meant  tasking  all  robots  with  the  same  action  (i.e.  waypoint,  play, 
superplay).  “Group”  robot  selection  was  also  possible  where  tasks 
could  be  given  to  groups  of  robots  smaller  or  equal  to  that  of  the 
whole  team. 


Figure  2  depicts  the  experimental  conditions  and  the  two 
dimensions  of  the  delegation  interface,  robot  selection  (on  the  X 
axis)  and  automation  level  (on  the  Y  axis).  A  specific  X,Y 
coordinate  represents  an  interface  condition  that  is  either 
restricted  or  flexible  along  those  two  dimensions.  A  restricted 
interface  is  constrained  (i.e.  non-selectable)  to  a  specific  level 
along  each  dimension.  A  flexible  interface  is  either  not 
constrained  or  is  constrained  on  only  one  dimension. 


Robot  Selection 


s 

! 


Less  More 


1 . 

Individual 

Group 

3 

waypoint 

o 

o 

o 

P  lay 

© 

o 

o 

More 

Super- 

Play 

X 

o 

o 

Selectable 

o 

o 

o 

a 

© 

© 

o 

© 


Figure  2.  Interface  combinations  (conditions  1-5) 


For  example,  the  combination  of  “Individual- Waypoint”, 
identified  as  Coordinate  1  in  Figure  2,  represents  a  restricted 
interface  condition  -  Figure  3  shows  the  interface  presentation  for 
an  operator.  In  this  condition  participants  are  required  to 
supervise  individual  robots  using  only  waypoint  movements;  no 
other  options  are  available. 


Figure  3.  Individual  Waypoint  Interface  Condition 

The  combination  of  “Selectable-Play”,  identified  as  Coordinate  4 
in  Figure  2,  represents  a  flexible  interface  condition  -  Figure  4 
shows  the  interface  presentation  for  an  operator.  In  this  condition 
participants  can  flexibly  choose  how  to  select  robots,  whether  an 
individual,  group,  or  all;  however,  the  flexibility  is  limited 
because  they  are  constrained  to  using  only  plays. 


Figure  4.  Selectable  Play  Interface  Condition 

As  shown  in  Figure  2,  five  different  interface  conditions  were 
selected:  (1)  individual  waypoint,  (2)  individual  play,  (3) 
selectable  waypoint,  (4)  selectable  play,  and  (5)  selectable 
selectable.  Conditions  1-2  were  selected  as  representative 
examples  for  restricted  interfaces.  Conditions  3-5  were  selected 
representative  flexible  interfaces  with  condition  5,  representing 
the  optimal  (i.e.  most  flexible)  delegation  interface. 

These  conditions  were  combined  factorially  with  the  number  of 
robots  controlled  by  the  participant  (four,  six,  or  eight  robots). 
Interface  condition  was  treated  as  a  blocked  factor  while  the 
number  of  robots  controlled  was  randomized  within  each  block. 
Each  participant  completed  two  sessions  (sessions  were  conducted 
on  the  same  day),  one  session  consisting  of  three  blocks,  and  the 
other  two  blocks.  Session  and  condition  were  counterbalanced  to 
offset  order  effects.  A  single  block  had  a  total  of  15  trials  (five 
trials  with  4  robots,  five  with  6  robots,  and  five  with  8  robots)  for 
a  total  of  75  trials.  Participants  completed  mental  workload  and 
situation  awareness  ratings  (0  (low)  to  100  (high))  after  each  trial, 
similar  to  the  NASA-TLX  [5]  and  the  3-D  SART  [14]  subjective 


28 


measure  questionnaires.  Robotic  usage  measures  and  performance 
data  were  also  logged  for  each  trial. 

Two  strategy  actions,  offensive  or  defensive,  were  defined.  During 
waypoint  (i.e.  point-to-point  clicking)  interface  conditions,  an 
offensive  or  defensive  task  was  defined  by  the  end-point  (i.e. 
where  the  robot  was  directed  to  move)  location.  If  the  robot 
location  at  which  the  action  was  implemented  (i.e.  x  and  y 
coordinates)  was  on  the  offensive  side  or  opponent  area  (as  seen 
in  Figure  1,  the  mid-line  distinguished  the  participant’s  area  (left) 
and  opponent’s  area  (right)),  then  the  action  was  defined  as  an 
offensive  task.  If  the  location  was  on  the  defensive  side  of  the 
playing  field  then  it  was  categorized  as  a  defensive  task.  During 
interface  conditions  in  which  participants  could  use  plays,  an 
offensive  task  was  defined  as  the  selection  of  an  offensive  play 
(e.g.  circle  offense).  A  defensive  task  was  defined  as  the  selection 
of  a  defensive  play  (e.g.  circle  defense). 

A  strategy  switch  occurred  when  the  participant  switched  from  an 
offensive  to  a  defensive  strategy  or  from  a  defensive  to  an 
offensive  task.  In  contrast,  a  task  was  considered  a  no-switch  task 
when  the  same  task  was  repeated  (e.g.  offensive,  offensive, 
offensive).  For  each  trial,  grand  averages  were  computed  for 
switch  and  no-switch  tasks  by  summing  the  times  taken  for  each 
task  action  and  dividing  by  the  number  of  actions  for  each.  Since 
the  first  task  action  performed  during  each  trial  did  not  have  a 
previous  task  action  to  which  it  would  be  referenced  those  times 
were  not  included  in  the  computations  (see  Table  1  for  example 
computations,  and  Figure  5  for  illustration). 


Table  1.  Computation  of  mean  switch  and  no-switch  times 


ActionfAction  Type) 

Total 

Time 

Switch 

Time 

No-Switch 

Time 

Defensive  (First  Action) 

2.17 

- 

- 

Defensive  (No-Switch) 

3.83 

- 

1.66 

Defensive  (No-Switch) 

4.5 

- 

0.67 

Offensive  (Switch) 

6 

1.5 

- 

Offensive  (No-Switch) 

6.57 

- 

0.57 

Defensive  (Switch) 

9.7 

3.13 

- 

Type  (NS  =  No-Switch  Time ,  S  =  Switch  Time) 

Strategy  (D  =  Defense,  O  =  Offense ) 

Type  N/A  NS  NS  S  NS  S 

♦ - • - 0 - • - 0 - 0 - ♦ 

Strategy  D  D  D  O  O  D 

Time  (s)  2.17  1.66  0.67  1.5  0.57  3.13 

Figure  5.  Illustration  of  Strategy  Usage  as  defined  by  Switch 
and  No-Switch  Actions 


Apparatus  and  Procedures 

The  RoboFlag  simulation  ran  on  two  separate  PCs  communicating 
under  TCP/IP  protocol.  The  participant  used  one  PC  while 
another  PC  ran  the  opposing  team  script,  and  recorded  log  files. 
To  observe  participants’  actions  unobtrusively,  an  additional 
monitor  was  used  by  the  experimenter  for  observation.  The  basic 
RoboFlag  objectives  and  architecture  were  unchanged  from 
previous  studies  [13],  but  changes  made  to  the  RoboFlag 
environment  are  described  below.  Of  the  five  experimental 
conditions  selected,  two  represented  restricted  interfaces 
(individual  waypoint,  individual  play),  and  three  flexible 
interfaces  (selectable  waypoint,  selectable  play,  and  selectable 
selectable).  The  opponent  force  consisted  of  6  robots  that  were 
scripted  to  have  a  mixed  “posture”,  defending  their  flag  half  the 
time  and  going  on  offense  during  the  remaining  half.  When  the 
opponent  team  conducted  offensive  maneuvers,  the  attacking 
routes  of  the  robots  were  varied  in  an  unpredictable  manner. 

Participants  received  instruction  (but  with  no  specific  strategy 
guidance)  on  interface  features  and  procedures  for  robot  selection 
and  movement.  Three  training  trials  were  conducted  for  each 
condition  for  each  robot  number  (4,  6,  and  8).  During 
experimental  trials,  participants  were  told  the  number  of  robots 
before  each  trial.  A  trial  concluded  when  either  the  participant  or 
opponent  had  captured  the  flag  and  crossed  the  mid-line  to  their 
side. 

3.  RESULTS 

Given  the  limited  space  for  this  paper,  results  are  reported  only 
for:  (1)  overall  mission  execution  time,  (2)  Fitt’s  law  movement 
calculations,  (3)  number  of  strategies  used  during  a  trial,  and  (4) 
switch  /  no-switch  results.  Mission  execution  time  data  were 
submitted  to  an  ANOVA  and  resulted  in  a  main  effect  for 
Interface  condition,  F(4,44)  =  4.29,  p  <  .01.  A  planned 
comparison  between  the  manual  only  (individual  waypoint  and 
selectable  waypoint)  and  automated  only  conditions  (individual 
play  and  selectable  play)  revealed  a  significant  effect  t(ll),  =  - 
4.60,  p  <  .001,  with  mission  execution  time  significantly  longer 
for  the  automated  only  condition,  than  the  manual  only  condition 
(see  Figure  6).  This  result  illustrates  the  same  general  effect 
obtained  by  Parasuraman  et  al.  [13],  When  participants  were 
restricted  to  the  use  of  automated  plays  only,  overall  human-robot 
performance  was  less  efficient.  Moreover,  when  operators  could 
flexibly  use  either  waypoint  control  or  automated  plays  (selectable 
all),  mission  time  was  reduced,  although  in  this  study  this  was 
only  a  marginally  significant  effect. 


80 


Individual  Selectable  Individual  Selectable  Selectable 
Waypoint  Waypoint  Play  Play  Selectable 


Interface  Type 

Figure  6.  Mission  Time  for  Interface  Conditions 
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We  also  analyzed  the  time  it  took  operators  to  switch  from  one 
type  of  strategy  to  the  other  (e.g.,  offensive  to  defensive),  and 
compared  these  to  cases  where  participants  implemented  strategy 
actions  that  did  not  switch  (e.g.,  offensive  to  offensive).  Because 
the  physical  action  for  implementing  a  strategy  was  a  mouse  click, 
the  motor  movement  time  in  moving  the  cursor  needed  to  be 
assessed  before  switch  and  no-switch  times  could  be  analyzed. 
Fitts’  Law  was  used  to  calculate  movement  time  -  in  this  case  the 
time  required  to  move  the  cursor  from  one  position  on  the 
graphical  user  interface  (GUI)  to  another  position.  The  specific 
version  of  Fitts’  Law  known  as  Shannon’s  formulation  was  used  - 
MT  =  a  +  b  log2(2A/W  +  1 ), where  MT  is  the  movement  time,  a,b 
are  the  regression  coefficients  for  a  particular  user  and  mouse,  A  is 
the  distance  of  the  movement  from  start  to  target  center,  and  W  is 
the  width  of  the  target.  The  regression  coefficients  used  for  point 
and  click  mouse  moves  were  taken  from  Mackenzie’s,  POINT- 
SELECT  [7).  The  GUI  movement  times  for  the  best  and  worst 
case  are  provided  in  Table  2.  The  best  case  GUI  movement  time 
represented  a  situation  where  the  participant  made  a  very  small 
motor  movement,  such  as  when  the  operator  changed  a  play 
without  having  to  de-select  the  robot.  For  example,  if  the  operator 
switched  from  a  circle  offense  play  (top  most  play  in  Figure  1)  to 
a  circle  defense  play  then  the  operator  only  needed  to  move  0.80 
inches  on  the  GUI.  If  however,  the  operator  selected  a  robot  at  the 
left  most  bottom  comer  of  the  playing  field,  and  then  selected 
circle  offense,  the  action  required  a  motor  movement  of  6.4 
inches.  This  represented  a  worst  case  situation. 


Table  2.  Best  and  Worst  Case  Movement  Time  Predictions 


Fitts'  Law 
Calculations 

Total  (seconds) 

A  value 
(inches) 

W  value 
(inches) 

Best 

0.43 

0.80 

0.63 

Worst 

0.81 

6.40 

0.63 

This  analysis  indicates  that  the  average  movement  time  to  execute 
a  strategy  change  was  relatively  short  (~  0.6  s  on  average).  As 
shown  later,  the  computed  switch  times  were  considerably  longer 
(from  2  to  7  s)  than  the  mean  motor  movement  time.  Therefore  the 
contribution  of  changes  in  motor  movement  time  to  changes  in 
switch  time  with  changes  in  the  interface  are  likely  to  have  been 
small. 

We  next  analyzed  changes  in  the  number  of  strategy  actions  for 
the  different  interface  conditions  and  number  of  robots.  The 
absolute  counts  for  number  of  actions  performed  were  submitted 
to  a  2  X  5  X  3  analysis  of  variance  (ANOVA)  with  factors  of 
strategy  switch  type  (switch,  no-switch),  interface  condition 
(individual  waypoint,  individual  play,  selectable  waypoint, 
selectable  play,  and  selectable  selectable),  and  number  of  robots 
(4,6,8).  There  was  a  significant  main  effect  for  strategy  type, 
F(l,ll)  =  24.32,  p  <  0.01.  Indicating  that  significantly  more  no¬ 
switch  strategy  actions  ( M  =  21.57,  SE  =  3.58)  than  switch 
strategy  actions  ( M  =  3.58,  SE  =  0.28)  were  performed.  There 
was  also  a  significant  main  effect  of  interface  condition  on  actions 
performed,  F(4,44)  =  5.35,  p  <  0.05.  As  illustrated  in  Figure  7 
most  of  the  variation  between  the  number  of  actions  performed 
seems  to  be  due  to  the  no-switch  rather  than  the  switch  strategy 
actions. 


Individual  Selectable  Individual  Selectable  Selectable 
Manual  Manual  Play  Play  Selectable 


Interface  Condition 

Figure  7.  Number  of  Actions  for  continuous  and  switch  tasks 
by  Interface  Conditions. 

The  only  other  significant  effect  obtained  was  for  the  number  of 
robots,  F( 2,22)  =  3.90,  p  <  0.05.  Post  hoc  pairwise  comparisons 
with  a  Bonferroni  adjustment  gave  a  significant  effect  for  robot 
number,  p  <  0.05;  participants  performed  more  actions  with  8 
robots  (M  =  13.90  actions,  SE  =  2.13)  than  with  4  robots  (M  = 
1 1.67,  SF“  1.93)  [see  Figure  8], 


4  6  8 

Num  ber  of  Robots 


Figure  8.  Number  of  Actions  for  Number  of  Robots. 


To  assess  the  changes  in  switch  and  no-switch  times  with 
interface  type  and  robot  number,  a  2  X  5  X  3  ANOVA  was 
computed  with  the  same  factors  as  before.  Significant  main  effects 
were  obtained  for  task  type,  F(l,ll)  =  78.50,  p  <  0.01  and 
interface  condition,  F(4,44)  =  20.13,  p  <  0.01.  The  only  other 
significant  effect  was  the  interaction  between  strategy  type  and 
interface  condition,  F(4,44)  =  10.88,  p  <  0.01  [see  Figure  9],  As 
expected,  participant’s  times  were  longer  for  switch  strategies  ( M 
=  4.59  s,  SE  =  0.29  s)  than  for  no-switch  strategies  ( M  =  1.87  s, 
SE  =  0.18  s). 
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Figure  9.  Switch  and  No-Switch  times  for  Interface 
Conditions. 

To  examine  the  hypothesis  that  switch  cost  effects  were  lower  for 
manual  than  automated  conditions,  pairwise  comparisons  were 
conducted  between  the  manual,  automated,  and  selectable 
selectable.  As  expected  there  were  no  significant  differences 
obtained  for  no-switch  strategies  (p  >  0.02,  a  =  0.05/3),  and  three 
significant  differences  for  switch  strategies.  The  manual  switch 
task  (M  =  2.47  s,  SE  =  0.26  s)  was  faster  than  the  automated 
switch  task  (M  =  6.55  s,  SE  =  0.44  s),  and  selectable  selectable 
switch  task  ( M  =  4.40  s,  SE  =  0.62  s),  /(1 1)  =  8.50,  p  <  0.01  and 
r(l  1)  =  3.16,  p  <  0.01  respectively.  In  addition  the  selectable 
selectable  switch  task  action,  while  slower  than  the  manual  switch 
task,  was  significantly  faster  than  the  automated  switch  task,  r(l  1) 
=  3.45,  p  <  0.01  [see  Figure  10],  These  findings  indicate  actions 
or  tasks  that  are  more  specific  in  nature  (e.g.  waypoint  action)  are 
less  complex  and  therefore  reduce  the  switch  cost  times  associated 
with  those  tasks.  However,  automated  tasks  appear  to  add 
additional  complexity  (e.g.  an  operator  has  to  consider  the  impact 
of  an  automated  play  with  multiple  tasks  associated  with  it)  that 
results  in  increased  switch  cost  times. 


Manual  Selectable  Automated 

Selectable 

Interface  Condition 


Figure  10.  Switch  times  for  Manual,  Selectable  Selectable,  and 
Automated  Interface  Conditions. 


4.  DISCUSSION 

The  present  study  of  multi-robot  control  using  the  RoboFlag 
simulation  examined  the  effects  of  interface  type  and  number  of 
robots  controlled  on  overall  mission  execution  time  as  well  as  task 


switching  times.  Consistent  with  our  previous  findings  [13], 
restricting  participants  to  the  use  of  an  interface  that  allowed  only 
automated  plays  was  associated  with  longer  mission  completion 
times.  At  the  same  time,  there  was  a  trend  for  reduced  mission 
completion  times  when  participants  could  flexibly  use  either 
waypoint  control  or  automated  plays,  providing  general  support 
for  the  efficacy  of  delegation  interfaces. 

We  also  examined  switch  times  for  instances  when  participants 
implemented  a  different  tasking  action,  e.g.,  from  offense  to 
defense,  on  a  robot  or  group  of  robots.  The  results  showed  that 
such  switch  times  were  relatively  long,  on  the  order  of  2-7  s. 
These  delays  represent  a  significant  switch  cost,  because  switch 
times  were  significantly  longer  than  no-switch  times  (1-2  s),  when 
participants  initiated  successive  robot  tasking  actions  that  were 
the  same  (e.g.,  both  offensive),  and  substantially  longer  than  the 
mean  motor  movement  time  required  to  execute  the  action  using  a 
mouse  click  (~  0.5  s).  These  values  for  switch  costs  are 
comparable  to  those  reported  in  previous  studies  with  complex 
multi-task  [2]  and  robot  simulations  [6]. 

The  results  supported  the  general  hypothesis  that  switch  costs 
would  vary  with  interface  type.  The  range  of  variation  was 
considerable  (2-7  s),  indicating  that  interface  type  could  be  a 
major  limiting  factor  in  human  supervision  of  multiple  robots, 
particularly  in  high-tempo  operations.  Given  the  goal  of  many 
programs  of  having  a  single  operator  supervise  as  many  robots  as 
possible,  these  findings  indicate  that  single  operator  control  of 
many  robots  may  not  be  feasible  for  certain  interface  types. 
Second,  it  was  hypothesized  that  switch  costs  would  be  greater 
when  participants  had  to  use  automated  plays  than  for  interaction 
at  a  lower  of  automation,  as  in  waypoint-to-waypoint  control.  The 
results  supported  this  prediction  as  well.  Whereas  mean  switch 
time  for  the  individual  manual  condition  was  about  2  s,  switch 
time  for  the  individual  play  condition  was  about  6  s,  a  three-fold 
increase.  The  increase  in  switch  time  for  more  automated  levels  of 
interaction  with  multiple  robots  is  also  consistent  with  the 
findings  of  Crandall  et  al  [4],  who  compared  teleoperated, 
waypoint-to-waypoint,  and  multiple-waypoint  methods  of  robot 
interaction  in  a  simulated  navigation  task. 

In  addition,  we  hypothesized  that  a  flexible  delegation  interface, 
where  participants  could  choose  between  different  task  levels, 
would  reduce  switch  costs  compared  to  restricted  interfaces.  This 
prediction  was  partially  supported.  In  our  previous  study  [13],  we 
found  that  when  participants  were  provided  the  ability  to  delegate 
(or  not)  task  functions  to  automation,  they  benefited  from  being 
able  to  use  manual  control  flexibly  with  automation  to  reduce 
their  overall  mission  execution  time.  Likewise,  when  participants 
in  the  present  study  were  provided  the  ability  to  delegate,  they 
received  a  similar  benefit  in  the  form  of  a  reduced  switch  cost, 
from  6  s  (in  the  individual  play  condition)  to  about  4  s  (in  the 
selectable  selectable  condition). 

The  task  switching  measure  used  in  the  present  study,  following 
the  procedure  developed  by  [2]  provides  a  framework  for 
exploring  timing  differences  associated  with  the  use  of  different 
interfaces  for  controlling  multiple  UVs.  The  metric  provided 
useful  information  on  the  relative  merits  of  different  delegation 
interfaces  beyond  that  gained  from  analysis  of  overall  mission 
completion  time.  Future  research  should  seek  to  investigate  more 
specifically  the  characteristics  of  task  switching  in  human-robot 
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interaction,  and  to  examine  how  task  switch  costs  impact  global 

measures  such  as  mission  execution  time  or  mission  success. 
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